Traveling road surface detection apparatus and traveling road surface detection method

ABSTRACT

An apparatus that detects a road surface on the basis of a parallax image around a vehicle, the road surface detection apparatus includes an ECU configured to determine on the basis of parallax information of the parallax image whether a unit area is a road surface area for each unit area of the parallax image associates each unit area with a grid in a map on the basis of parallax information and coordinate position of the unit area, an eye point of the map being set so as to be higher than an eye point of the parallax image, the map being obtained by partitioning an area around the vehicle in a grid, and detect road surface grids and each grid located between the road surface grids as the road surface, the road surface grids being the grids corresponding to the unit areas determined as the road surface areas.

INCORPORATION BY REFERENCE

The disclosure of Japanese Patent Application No. 2015-023400 filed onFeb. 9, 2015 including the specification, drawings and abstract isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a traveling road surface detection apparatusand a traveling road surface detection method.

2. Description of Related Art

U.S. Patent Application Publication No. 2014/0071240 describes anapparatus that detects a traveling road surface. This apparatus acquiresa parallax image, in which parallax information is associated with eachpixel, by the use of images respectively obtained from right and leftcameras, identifies a road surface area on the basis of the pieces ofparallax information of the parallax image, and performs coordinatetransformation of the parallax image onto a grid plan map, thusdetecting a traveling road surface.

However, at the time of coordinate transformation from the parallaximage onto the plan map, a continuous road surface area in the parallaximage may be discontinuous areas on the grid plan map because of, forexample, the difference between the eye point of the parallax image andthe eye point of the plan map. In this case, the apparatus described inU.S. Patent Application Publication No. 2004/0071240 erroneously detectsthe continuous traveling road surface as a traveling road surfaceincluding multiple gaps or divided multiple traveling road surfaces. Inthis technical field, a traveling road surface detection apparatus and atraveling road surface detection method that are able to improve adecrease in the accuracy of detecting a traveling road surface aredesired.

SUMMARY OF THE INVENTION

An aspect of the invention provides a traveling road surface detectionapparatus that detects a traveling road surface on the basis of aparallax image around a vehicle. The parallax image is acquired by anin-vehicle camera. The traveling road surface detection apparatusincludes an electronic control unit configured to i) determine on thebasis of parallax information of the parallax image whether a unit areais a road surface area or a non-road surface area for each unit area ofthe parallax image, each unit area including one or multiple pixels, ii)associate each unit area with a grid in a map on the basis of parallaxinformation, and coordinate position of the unit area, an eye point ofthe map being set so as to be higher than an eye point of the parallaximage, the map being obtained by partitioning an area around the vehiclein a grid, and iii) detect road surface grids and each grid locatedbetween the road surface grids on the map as the traveling road surfaceon the map, the road surface grids being the grids corresponding to theunit areas determined as the road surface areas.

With this traveling road surface detection apparatus, after coordinatetransformation of the parallax image onto the map is performed by theelectronic control unit, each grid that is not a road surface grid butlocated between road surface grids on the map is detected as thetraveling road surface on the map by the electronic control unit.Therefore, with this traveling road surface detection apparatus, forexample, even when areas on the map corresponding to the road surfaceareas are discontinuous at the time when the parallax image includingthe road surface areas is subjected to coordinate transformation andprojected onto the map, it is possible to detect the traveling roadsurface by interpolating the discontinuous areas. Therefore, thisapparatus is able to improve a decrease in the accuracy of detecting atraveling road surface.

In one embodiment, the electronic control unit may detect the travelingroad surface on the map by setting each grid sandwiched between the roadsurface grids arranged in one of radially extending directions withreference to a position of the in-vehicle camera on the map for the gridlocated between the road surface grids on the map. For example, if theapparatus is configured to detect each grid sandwiched between the roadsurface grids arranged in a lateral direction (vehicle width direction)with reference to the position of the in-vehicle camera on the map asthe traveling road surface, there is a concern that the grid behind anobstacle, which cannot be recognized from the in-vehicle camera that isan observation point, is detected as the traveling road surface. Incontrast, the traveling road surface detection apparatus sets each gridsandwiched between the road surface grids arranged in the radiallyextending direction with reference to the position of the in-vehiclecamera as the grid located between the road surface grids on the map.Therefore, for example, it is possible to avoid erroneous detection ofthe grid behind an obstacle as the traveling road surface.

In one embodiment, the traveling road surface detection apparatus mayfurther include an area identification unit that, on the basis of acoordinate position or pixel information of each unit area determined asthe road surface area by the area determination unit, associatesidentification information that identifies the corresponding roadsurface area with each unit area determined as the road surface area bythe area determination unit. When the pieces of identificationinformation respectively associated with the unit areas corresponding toa pair of the road surface grids that sandwich the grid are differentfrom each other, the detection unit may not detect the grid sandwichedbetween the pair of the road surface grids as the traveling roadsurface. In this case, with the traveling road surface detectionapparatus, for example, when a plurality of road surface areas areincluded in the parallax image, it is possible to avoid erroneousdetection of the grid corresponding to each unit area between the roadsurface areas as the traveling road surface.

Another aspect of the invention provides a traveling road surfacedetection method that detects a traveling road surface on the basis of aparallax image around a vehicle. The parallax image is acquired by anin-vehicle camera. The traveling road surface detection method includes:determining on the basis of parallax information of the parallax imagewhether a unit area is a road surface area or a non-road surface areafor each unit area of the parallax image, each unit area including oneor multiple pixels; associating each unit area with a grid in a map onthe basis of parallax information and coordinate position of the unitarea, an eye point of the map being set so as to be higher than an eyepoint of the parallax image, the map being obtained by partitioning anarea around the vehicle in a grid; and detecting road surface grids andeach grid located between the road surface grids on the map as thetraveling road surface on the map, the road surface grids being thegrids corresponding to the unit areas determined as the road surfaceareas. With this method, as well as the advantageous effects obtainedfrom the traveling road surface detection apparatus according to theabove-described aspect of the invention, it is possible to improve adecrease in the accuracy of detecting a traveling road surface.

According to various aspects and embodiments of the invention, it ispossible to improve a decrease in the accuracy of detecting a travelingroad surface.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, advantages, and technical and industrial significance ofexemplary embodiments of the invention will be described below withreference to the accompanying drawings, in which like numerals denotelike elements, and wherein:

FIG. 1 is a block diagram that illustrates a traveling road surfacedetection apparatus according to an embodiment;

FIG. 2 is a view that shows an example of a parallax image captured by astereo camera;

FIG. 3A is a table that illustrates an example of data that areprocessed by the traveling road surface detection apparatus according tothe embodiment;

FIG. 3B is a table that illustrates an example of data that areprocessed by the traveling road surface detection apparatus according tothe embodiment;

FIG. 3C is a table that illustrates an example of data that areprocessed by the traveling road surface detection apparatus according tothe embodiment;

FIG. 3D is a table that illustrates an example of data that areprocessed by the traveling road surface detection apparatus according tothe embodiment;

FIG. 4 is an example of road surface areas in the parallax image;

FIG. 5A is a view that shows an example of a predetermined area in theparallax image in an example of coordinate transformation from theparallax image to a grid map;

FIG. 5B is a view that illustrates an example of a grid mapcorresponding to the predetermined area of the parallax image as anexample of coordinate transformation from the parallax image to the gridmap;

FIG. 6A is a view that illustrates an example in the case where the gridmap is scanned in radially extending directions with reference to theposition of the stereo camera as an example of a traveling road surfacedetection process on the grid map;

FIG. 6B is a view that illustrates the traveling road surface detectionprocess on the grid map shown in FIG. 6A;

FIG. 7 is a flowchart that shows an example of a traveling road surfacedetection method that is used by the traveling road surface detectionapparatus according to the embodiment; and

FIG. 8 is a flowchart that shows an example of the process of outputtingtraveling road surface information by the traveling road surfacedetection apparatus according to the embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the invention will be described withreference to the accompanying drawings.

FIG. 1 is a block diagram that illustrates a traveling road surfacedetection apparatus according to the present embodiment. The travelingroad surface detection apparatus 1 shown in FIG. 1 detects a travelingroad surface on the basis of a parallax image around a vehicle. Thetraveling road surface is, for example, a road surface on which thevehicle is allowed to travel. The traveling road surface may include notonly the road surface of a road on which the vehicle travels but alsothe road surface of a passage and parking space in a parking lot. In thepresent embodiment, description will be made on the assumption that thetraveling road surface detection apparatus 1 is mounted on a vehicle,such as a passenger automobile.

The traveling road surface detection apparatus 1 detects a travelingroad surface on the basis of a parallax image acquired by an in-vehiclecamera that captures images around the vehicle. The parallax image is animage (data) that includes parallax information or depth information.For example, parallax information or depth information is associatedwith each of pixels that constitute the parallax image. Morespecifically, the parallax image is an image in which, for example,coordinate positions in the image are associated with the pieces ofparallax information or depth information.

Configuration of Traveling Road Surface Detection Apparatus

As shown in FIG. 1, the traveling road surface detection apparatus 1includes an electronic control unit (ECU) 2 and a stereo camera 3 (anexample of an in-vehicle camera) for the purpose of detecting atraveling road surface. The ECU 2 is an electronic control unitincluding a central processing unit (CPU), a read only memory (ROM), arandom access memory (RAM), a controller area network (CAN)communication circuit, and the like. The ECU 2 is, for example,connected to a network through which communication is carried out withthe use of the CAN communication circuit, and is communicably connectedto the stereo camera 3. The ECU 2, for example, inputs or outputs databy operating the CAN communication circuit on the basis of signalsoutput from the CPU, stores input data in the RAM, loads programs storedin the ROM onto the RAM, and executes the programs loaded onto the RAM,thus implementing the functions of components (described later). The ECU2 may be formed of a plurality of electronic control units.

The stereo camera 3 is an image acquisition device that acquires aparallax image by capturing images around the vehicle. The stereo camera3 includes, for example, two cameras. In FIG. 1, the stereo camera 3includes a first camera 4 and a second camera 5 arranged so as toreplicate a binocular parallax. The first camera 4 and the second camera5 are, for example, provided on the back side of the windshield of thevehicle, and capture images ahead of the vehicle.

The stereo camera 3 generates a parallax image by using a first imageand a second image respectively captured by the first camera 4 and thesecond camera 5. The stereo camera 3, for example, executes the processof searching the second image for corresponding points corresponding topixels of the first image with reference to the first image for eachpixel of the first image, and calculates parallax information for eachpixel.

The parallax information is, for example, a distance between a pixelposition of the first image and a pixel position of a correspondingpoint of the second image (a pixel-to-pixel distance in the image). Thestereo camera 3, for example, generates an image that associates anabscissa position x and an ordinate position y in the image with apixel-to-pixel distance as a parallax image. The pixel-to-pixel distanceis parallax information. The stereo camera 3 may obtain depthinformation (information about a distance from the stereo camera 3) foreach pixel by applying the parallax information of each pixel to ageneral formula for transformation, and may generate an image in whichan abscissa position x and an ordinate position y in the image areassociated with depth information, as a parallax image. The generalformula for transformation is a formula in which basic information (forexample, a camera-to-camera distance, and the like) of the stereo camera3 is set as a coefficient and parallax information and depth informationare inversely proportional to each other. The stereo camera 3 transmitsthe acquired parallax image to the ECU 2.

Next, the functional configuration of the ECU 2 will be described. Asshown in FIG. 1, the ECU 2 includes an image acquisition unit 10, anarea determination unit 11, an area identification unit 12, a coordinatetransformation unit 13 and a traveling road surface information outputunit (detection unit) 14.

The image acquisition unit 10 acquires a parallax image from the stereocamera 3. The image acquisition unit 10 is implemented when the CANcommunication circuit operates on the basis of a signal output from theCPU. FIG. 2 is a view that shows an example of a parallax image capturedby the stereo camera 3. The parallax image G1 shown in FIG. 2 is animage generated from captured images ahead of the vehicle. As shown inFIG. 2, a traveling road surface R on which the vehicle travels liesbetween a curb K1 and a guard rail K2. A sidewalk H1 lies between thecurb K1 and trees K3 located outside the curb K1. A sidewalk H2 liesbetween the guard rail K2 and buildings K4 located outside the guardrail K2. FIG. 3A is an example of a data table of a parallax image. Inthe table shown in FIG. 3A, a pixel ID that identifies each of pixels,an x-coordinate position in the parallax image G1, a y-coordinateposition in the parallax image G1 and depth information z are associatedwith one another. For example, (1,1) as the coordinates (x,y) and “100”as the depth information z are associated with the pixel ID “1”.Similarly, (1,2) as the coordinates (x,y) and “200” as the depthinformation are associated with the pixel ID “2”. Association of eachpixel ID with a pixel value is omitted here. The image acquisition unit10 stores the acquired parallax image in a storage area, such as theRAM.

The area determination unit 11 determines whether a unit area is a roadsurface area or a non-road surface area for each unit area of theparallax image G1 on the basis of the pieces of parallax information ofthe parallax image G1. Each unit area of the parallax image G1 includesone or multiple pixels. The area determination unit 11 is implementedwhen the CPU loads a program stored in the ROM onto the RAM and executesthe program loaded onto the RAM. The unit area is an area in theparallax image G1, and the area includes one or multiple pixels. Thatis, the minimum unit area is an area including one pixel. Hereinafter,an example in which the unit area is one pixel (per pixel) will bedescribed.

The area determination unit 11, for example, repeatedly executes theprocess of setting a pixel on which determination is carried out and theprocess of carrying out determination on the pixel by scanning theparallax image G1 from the top left pixel to the bottom right pixel. Theroad surface area is a range in which a road surface is drawn in theparallax image G1. The non-road surface area is an area other than theroad surface area, and is, for example, a range in which an object otherthan the road surface is drawn in the parallax image G1. The objectother than the road surface is, for example, a building, a tree, a guardrail, a curb, or the like.

An example of determination as to whether the pixel is a road surfacearea by the area determination unit 11 will be described. The areadetermination unit 11 estimates a road surface gradient for each pixelby utilizing statistical data acquired in advance. The statistical dataindicate the relationship between depth information and a road surfacegradient. Subsequently, the area determination unit 11 acquires heightinformation Hy (corresponding to y coordinate in the parallax image) ofthe estimated road surface gradient by using the depth information ofeach pixel (x coordinate, y coordinate) and the estimated correspondingroad surface gradient. The area determination unit 11 compares the ycoordinate with the height information Hy. When the difference issmaller than a predetermined value, the area determination unit 11determines that the pixel is a road surface area; whereas, when thedifference is larger than or equal to the predetermined value, the areadetermination unit 11 determines that the pixel is a non-road surfacearea. FIG. 3B is an example of a data table that is managed by the areadetermination unit 11. In the table shown in FIG. 3B, a pixel ID, a roadsurface flag that determines whether the pixel is a road surface area ora non-road surface area and a label (described later) are associatedwith one another. The road surface flag is data that indicates that thepixel is a road surface area when the road surface flag is set to “1”and that indicates that the pixel is a non-road surface area when theroad surface flag is “0”. The road surface flag is a result determinedby the area determination unit 11. For example, when the pixel of thepixel ID “1” is a road surface area, the area determination unit 11associates “1” as the road surface flag with the pixel ID “1”.Similarly, when the pixel of the pixel ID “2” is a non-road surfacearea, the area determination unit 11 associates “0” as the road surfaceflag with the pixel ID “2”.

On the basis of the coordinate position of each pixel determined as theroad surface area by the area determination unit 11, the areaidentification unit 12 associates identification information thatidentifies the corresponding road surface area with each pixeldetermined as the road surface area by the area determination unit 11.The area identification unit 12 is implemented when the CPU loads aprogram stored in the ROM onto the RAM and executes the program loadedonto the RAM. The identification information is, for example,information that is able to uniquely identify a road surface area group.A specific example of the identification information is a label thatindicates a road surface area group.

Initially, the area identification unit 12 determines connectivity withother pixels by using the coordinate position of each pixel determinedas the road surface area by the area determination unit 11. For example,when adjacent pixels in the upper, lower, right, left and obliquedirections around a pixel to be processed are pixels determined as roadsurface areas, the area identification unit 12 determines that there isconnectivity between the pixel to be processed and the adjacent pixels.Subsequently, the area identification unit 12 associates the same labelwith the pixel to be processed and the adjacent pixels, between which itis determined that there is connectivity. That is, when pixels areassociated with the same label, the pixels are included in the same roadsurface area. The above-described table shown in FIG. 3B is also managedby the area identification unit 12. When the area identification unit 12determines that there is connectivity between the pixel to be processedand the adjacent pixels, the area identification unit 12 updates thelabels of the table shown in FIG. 3B. For example, when the areaidentification unit 12 determines that there is connectivity between thepixel of the pixel ID “1” and the pixel of the pixel ID “4096”, the areaidentification unit 12 updates the labels of these pixel IDs with thesame label “2”. FIG. 4 is an example of road surface areas in a parallaximage. FIG. 4 shows three road surface areas R1, R2, R3 on the parallaximage G1. For example, the road surface area R1 consists of a group ofpixels associated with the label “1”, the road surface area R2 consistsof a group of pixels associated with the label “2”, and the road surfacearea R3 consists of a group of pixels associated with the label “3”.That is, pixels of the pixel IDs associated with the label “1” in thetable shown in FIG. 3B are pixels that constitute the road surface areaR1 in the parallax image G1 shown in FIG. 4. Similarly, pixels of thepixel IDs associated with the label “2” in the table shown in FIG. 3Bare pixels that constitute the road surface area R2 in the parallaximage G1 shown in FIG. 4. Similarly, pixels of the pixel IDs associatedwith the label “3” in the table shown in FIG. 3B are pixels thatconstitute the road surface area R3 in the parallax image G1 shown inFIG. 4.

The area identification unit 12 may determine whether there isconnectivity by using pieces of pixel information instead of thepositional relationship (positional coordinates) among pixels determinedas road surface areas. The pixel information is a pixel value, such as aluminance, a chromaticness and a contrast. For example, the areaidentification unit 12 may associate the same label with a group ofpixels among which a difference in luminance is smaller than or equal toa predetermined value. In this way, the area identification unit 12 isable to label pixels by using pixel values when the pieces of parallaxinformation of the pixels are not accurately acquired. The areaidentification unit 12 may determine whether there is connectivity byusing the positional relationship among pixels determined as roadsurface areas and the pieces of pixel information of the pixels.

The coordinate transformation unit 13 associates each pixel with a gridin a grid map (an example of a map) on the basis of the parallaxinformation and coordinate position of the pixel. The eye point of thegrid map is set so as to be higher than the eye point of the parallaximage G1. The grid map is obtained by partitioning an area spreadingahead of the vehicle (an example of an area around a vehicle) in a grid.The coordinate transformation unit 13 is implemented when the CPU loadsa program stored in the ROM onto the RAM and executes the program loadedonto the RAM.

The grid map is, for example, provided such that a two-dimensional planerepresented by a distance from the vehicle and a lateral position ispartitioned at set intervals into cells. The individual cells of thegrid map are called grids. Each cell may have a square shape or arectangular shape. The grid map has a data structure of, for example,data tables shown in FIG. 3C and FIG. 3D. As shown in FIG. 3C, theabscissa X (corresponding to the abscissa x of the parallax image) ofthe map and the ordinate Z (corresponding to the depth information z ofthe parallax image) of the map are associated with each grid ID thatuniquely identifies a corresponding one of the grids. The coordinatetransformation unit 13, for example, identifies a position on the gridmap, corresponding to each pixel ID, by substituting the correspondingparallax information and coordinate position, associated with the pixelID, into a general coordinate transformation formula, thus deriving thegrid ID corresponding to each pixel ID. That is, the coordinatetransformation unit 13 consults the data tables shown in FIG. 3A andFIG. 3B, and transforms the pixel position to the grid (X,Z) on the gridmap by using the positional information (x-coordinate position) and thedepth information z, associated with the pixel ID, and the coordinatetransformation formula, thus associating each pixel ID with acorresponding one of the grid IDs.

The coordinate transformation unit 13 assigns the road surface flag andlabel associated with each pixel ID to the grid ID corresponding to thepixel ID. For example, as shown in FIG. 3D, the road surface flag, thelabel and a search ID (described later) are associated with each gridID. The coordinate transformation unit 13 assigns the road surface flagand the label, associated with each pixel ID, to a corresponding one ofthe grid IDs. When one grid ID corresponds to a plurality of pixel IDs,the coordinate transformation unit 13 may determine one road surfaceflag and one label on the basis of a predetermined condition from theroad surface flags and labels associated with the plurality of pixelIDs, and may associate the determined road surface flag and label withthe grid ID. Hereinafter, description will be made on the assumptionthat a grid corresponding to a pixel determined as a road surface areaby the area determination unit 11, that is, a grid of which the roadsurface flag is “1”, is termed “road surface grid” and a grid of whichthe road surface flag is “0” is termed “non-road surface grid”. As willbe described later, because there is a grid that applies neither a roadsurface grid nor a non-road surface grid,

FIG. 3D shows an example in which the road surface flag of such a gridis set to a NULL value “−” (see the grid ID “2”).

FIG. 5A and FIG. 5B are views that illustrate an example of coordinatetransformation from the parallax image G1 to a grid map G2. FIG. 5A isan example of the parallax image G1. FIG. 5B is an example of the gridmap G2 corresponding to a predetermined area A1 in the parallax imageG1. The grid map G2 shown in FIG. 5B is a map in which an area ahead ofthe vehicle in plan view is partitioned in a grid, and data shown inFIG. 3C and FIG. 3D are visualized. The grid map G2 includes roadsurface grids RG. In the drawing, hatched grids are the road surfacegrids RG. In the drawing, the type of hatching is changed depending onthe label. For example, the label “1” is associated with the roadsurface grid RG corresponding to the road surface area R1, the label “2”is associated with the road surface grid RG corresponding to the roadsurface area R2, and the label “3” is associated with the road surfacegrid RG corresponding to the road surface area R3. That is, differenttypes of diagonal line hatching in the drawing are used respectively forthe label “1”, the label “2”, and the label “3”. In the grid map G2,non-road surface grids NG correspond to pixels associated with thenon-road surface areas. In the drawing, grids filled with dots are thenon-road surface grids NG.

The traveling road surface information output unit 14, for example,scans (searches) the grid map G2, connects the road surface grids RG inorder of the scanning direction, and detects an aggregate of theconnected road surface grids RG as a traveling road surface. Thetraveling road surface information output unit 14 is implemented whenthe CPU loads a program stored in the ROM onto the RAM and executes theprogram loaded onto the RAM. As shown in FIG. 5B, when the road surfaceareas R1, R2, R3 of the parallax image G1 are projected onto the gridmap G2, gaps appear in areas corresponding to the road surface areas R1,R2, R3 on the grid map G2. Gaps also appear in areas corresponding tothe non-road surface areas on the grid map G2. In FIG. 5B, gridssurrounded by thick-bordered boxes are gaps. Hereinafter, descriptionwill be made on the assumption that grids that do not correspond topixels associated with road surface areas or non-road surface areas aregap grids. The gap grids appear because of the difference between theeye point of the parallax image G1 and the eye point of the grid map G2,the difference in size between each pixel that is the constituent unitof the parallax image G1 and each grid that is the constituent unit ofthe grid map G2, the fact that the capturing accuracy of the stereocamera 3 decreases as a capturing distance increases and an error ofparallax information increases as a capturing distance increases, or thelike. When a traveling road surface has been detected by using the gridmap G2 that includes gap grids, there is a concern that the travelingroad surface that is originally continuous is erroneously detected asdivided traveling road surfaces.

Therefore, the traveling road surface detection apparatus 1 according tothe present embodiment, for example, detects a traveling road surface onthe assumption that an aggregate of road surface grids RG including gapsis an aggregate of road surface grids RG including no gaps.Specifically, the traveling road surface information output unit 14detects not only road surface grids RG but also gap grids EG locatedbetween the road surface grids RG on the grid map G2 as a traveling roadsurface on the grid map G2. Each gap grid EG between the road surfacegrids RG on the grid map G2 means the gap grid EG sandwiched between theroad surface grids RG in the scanning direction of the traveling roadsurface information output unit 14 on the grid map G2. For example, whenthe scanning direction (search direction) on the grid map G2 is the Xdirection (vehicle width direction, lateral direction), each gap grid EGlocated between the road surface grids RG on the grid map G2 is the gapgrid EG sandwiched between a pair of the road surface grids RG in theX-axis direction on the grid map G2. For example, when the scanningdirection (search direction) on the grid map G2 is the Y direction(depth direction, longitudinal direction), each gap grid EG locatedbetween the road surface grids RG on the grid map G2 is the gap grid EGsandwiched between a pair of the road surface grids RG in the Z-axisdirection on the grid map G2.

The traveling road surface information output unit 14 is able to scan(search) and connect road surface grids not only in the lateraldirection or the longitudinal direction but also in the obliquedirection, and detect a traveling road surface. For example, thetraveling road surface information output unit 14 may scan in theradially extending directions with respect to the position of the stereocamera 3 on the grid map G2. In this case, the gap grid EG sandwichedbetween the road surface grids RG means the gap grid EG sandwichedbetween a pair of the road surface grids RG arranged in one of theradially extending directions with reference to the position of thestereo camera 3.

The gap grid EG located between road surface grids RG on the grid map G2may mean the gap grid EG only when the gap grid EG is sandwiched betweena pair of adjacent road surface grids RG. As a specific example, onlywhen a road surface grid RG, a gap grid EG and a road surface grid RGare arranged sequentially in the scanning direction, the gap grid EG isregarded as the gap grid EG located between the road surface grids RG onthe grid map G2. By limiting the gap grid EG to the case where the gapgrid EG is sandwiched between adjacent road surface grids RG, forexample, even when a road surface grid RG has been erroneously detectedat a distant location, it is possible to suppress detection of an areathat should not be detected as a traveling road surface originally byusing the erroneously detected road surface grid RG. On the contrary,the gap grid EG located between road surface grids RG on the grid map G2is not limited to the gap grid EG only when the gap grid EG issandwiched between a pair of adjacent road surface grids RG, but the gapgrid EG located between road surface grids RG may mean a plurality ofgap grids EG sandwiched between a pair of road surface grids RG. In thiscase, for example, even when a road surface grid RG, a gap grid EG1, agap grid EG2 and a road surface grid RG are arranged sequentially in thescanning direction, the sandwiched two gap grids EG1, EG2 each are thegap grid EG located between road surface grids RG on the grid map G2. Bynot limiting the number of sandwiched gap grids EG, the opportunity todetect a gap grid EG as a traveling road surface increases, so it ispossible to further improve a decrease in the accuracy of detecting atraveling road surface. Alternatively, the gap grid EG located betweenroad surface grids RG on the grid map G2 may be limited to the casewhere the distance between a pair of road surface grids RG sandwichingthe gap grid EG is larger than or equal to a predetermined value. Inthis case, for example, when there is a low-height obstacle, such as acurb, and a road surface area is recognized behind the obstacle in theparallax image G1, it is possible to suppress detection of an area thatshould not be detected as a traveling road surface originally. Thetraveling road surface information output unit 14 is able to limit thegap grid EG located between road surface grids RG on the grid map G2 tothe above-described various meanings by setting the range of searchingfor one road surface grid RG to the next road surface grid RG to, forexample, a predetermined value or smaller or a predetermined value orlarger.

FIG. 6A and FIG. 6B are views that illustrate an example of a travelingroad surface detection process on the grid map G2. FIG. 6A is an exampleof the case where the grid map G2 is scanned in the radially extendingdirections with reference to the position of the stereo camera 3, andeach of the arrows Y indicates any one of the scanning directions. Asearch ID that uniquely identifies an arrow is assigned to each arrow.As shown in FIG. 3D, the traveling road surface information output unit14 assigns a search ID regarding a corresponding one of the arrows Y toeach of grid IDs of grids arranged along the corresponding one of thearrows Y shown in FIG. 6A. When the traveling road surface informationoutput unit 14 searches in each arrow direction in the drawing on thegrid map G2, the traveling road surface information output unit 14extracts grid IDs with which the same search ID as the search IDregarding the corresponding arrow is associated. The traveling roadsurface information output unit 14 virtually one-dimensionally arrangesthe extracted grid IDs in order of distance from the vehicle by the useof the positional information of each grid ID of FIG. 3C, and scans thegrids arranged one-dimensionally from the vehicle side toward thedistant side, thus scanning in the oblique direction.

An example of the traveling road surface detection process on the gridmap G2 will be described in detail by the use of the arrow Y shown inFIG. 6A and FIG. 6B. The search ID of the arrow Y is “1”. The travelingroad surface information output unit 14 extracts grid IDs “1”, “3”, . .. , “240” associated with the search ID “1” from the data table shown inFIG. 3D, and scans the grids of the extracted grid IDs in order ofdistance from the vehicle. FIG. 6B is a view that illustrates thetraveling road surface detection process on the grid map G2 shown inFIG. 6A. As shown in FIG. 6B, a portion in which a road surface gridRG1, a gap grid EG1, a road surface grid RG2, a non-road surface gridNG1, a gap grid EG2, and a road surface grid RG3 are arranged in thisorder will be mainly described. The labels of the road surface grid RG1and road surface grid RG2 are “1”, and the label of the road surfacegrid RG3 is “3”.

The traveling road surface information output unit 14 scans along thearrow Y, and determines that road surface grids are continuous from thevehicle to the road surface grid RG1. When the traveling road surfaceinformation output unit 14 has detected the gap grid EG1, the travelingroad surface information output unit 14 determines whether there is aroad surface grid RG on a further distant side. There is the roadsurface grid RG2 on the distant side of the gap grid EG1. The travelingroad surface information output unit 14 determines whether the label ofthe road surface grid RG1 and the label of the road surface grid RG2 arethe same. Because the labels of the road surface grid RG1 and roadsurface grid RG2 are “1”, the traveling road surface information outputunit 14 determines that the road surface grid RG1 and the road surfacegrid RG2 are the pair of road surface grids RG arranged in one of theradially extending directions, and determines that the gap grid EG1 is agap grid EG sandwiched between the pair of road surface grids RG. Thetraveling road surface information output unit 14 converts the gap gridEG1 to a road surface grid RG. Specifically, the traveling road surfaceinformation output unit 14 changes the road surface flag of the gap gridEG1 from “−” to “1” in the data tables shown in FIG. 3A to FIG. 3D. Thetraveling road surface information output unit 14 may assign the samelabel as the labels of the pair of road surface grids RG as the label ofthe gap grid EG1. Thus, an aggregate of the road surface grids RG with agap is interpolated.

On the other hand, after the traveling road surface information outputunit 14 determines that the road surface grids RG are continuous fromthe vehicle to the road surface grid RG2, the traveling road surfaceinformation output unit 14 further scans along the arrow Y. When thetraveling road surface information output unit 14 has detected thenon-road surface grid NG1, the traveling road surface information outputunit 14 determines whether there is a road surface grid RG on thefurther distant side. There are the gap grid EG2 and the road surfacegrid RG3 on the far side of the non-road surface grid NG1. The travelingroad surface information output unit 14 determines whether the label ofthe road surface grid RG2 is the same as the label of the road surfacegrid RG3. The label of the road surface grid RG2 is “1”, and the labelof the road surface grid RG3 is “3”. In this way, when the labelsrespectively associated with pixels corresponding to a pair of roadsurface grids RG that sandwich a gap grid EG are different from eachother, the traveling road surface information output unit 14 does notdetect the gap grid EG sandwiched between the pair of road surface gridsRG as a traveling road surface. That is, the traveling road surfaceinformation output unit 14 does not convert the gap grid EG2 to a roadsurface grid RG. More specifically, the traveling road surfaceinformation output unit 14 keeps the road surface flag of the gap gridEG at “−” in the data tables shown in FIG. 3A to FIG. 3D. In this way,the traveling road surface detection apparatus 1 is able to avoiderroneous detection of a gap grid EG, corresponding to a pixel betweenthe road surface areas R1, R3, as a traveling road surface when theplurality of road surface areas R1, R2, R3 are included in the parallaximage.

The traveling road surface information output unit 14 executes theabove-described process on all the search IDs. As shown in FIG. 6B, anaggregate of road surface grids RG is interpolated. The traveling roadsurface information output unit 14 selects an aggregate closest to thevehicle from among a plurality of aggregates of road surface grids RG.For example, in the situation shown in FIG. 6B, the traveling roadsurface information output unit 14 selects an aggregate RX of roadsurface grids RG. The traveling road surface information output unit 14detects the area of the aggregate RX of road surface grids RG as atraveling road surface on the grid map G2. In this way, when anaggregate of road surface grids RG includes a gap, the traveling roadsurface detection apparatus 1 detects a traveling road surface on theassumption that the aggregate of road surface grids RG includes no gap.

Traveling Road Surface Detection Method for Traveling Road SurfaceDetection Apparatus

Next, a traveling road surface detection method for the traveling roadsurface detection apparatus 1 will be described with reference to theaccompanying drawings. FIG. 7 is a flowchart that shows an example ofthe traveling road surface detection method that is used by thetraveling road surface detection apparatus 1. The flowchart shown inFIG. 7 is, for example, executed at preset time intervals while thevehicle is driven by an engine.

As shown in FIG. 7, the ECU 2 acquires an image with the use of theimage acquisition unit 10 in step S 101. The image acquisition unit 10acquires a parallax image around the vehicle from the stereo camera 3.

Subsequently, in step S102, the ECU 2 executes the process ofdetermining whether a pixel is a road surface area (area determinationstep) for each pixel (each unit area) by the area determination unit 11.The area determination unit 11 estimates a road surface gradient foreach pixel by utilizing statistical data acquired in advance andindicating the relationship between depth information and a road surfacegradient. Subsequently, the area determination unit 11 acquires theheight information Hy (corresponding to y coordinate in the parallaximage) of each estimated road surface gradient by using the depthinformation of the corresponding pixel (x coordinate, y coordinate) andthe estimated road surface gradient. The area determination unit 11compares the y coordinate with the height information Hy. When thedifference is smaller than a predetermined value, the area determinationunit 11 determines that the intended pixel is a road surface area;whereas, when the difference is larger than or equal to thepredetermined value, the area determination unit 11 determines that theintended pixel is a non-road surface area.

Subsequently, in step S103, the ECU 2 executes the process of labelingeach pixel (each unit area) with the use of the area identification unit12. The area identification unit 12 associates a label that identifiesthe corresponding road surface area with each pixel determined as theroad surface area by the area determination unit 11 on the basis of thecoordinate position of the pixel determined as the road surface area bythe area determination unit 11.

Subsequently, in step S104, the ECU 2 executes projection process(coordinate transformation process) onto the grid map with the use ofthe coordinate transformation unit 13. The coordinate transformationunit 13 associates each pixel with a grid in the grid map (an example ofa map) on the basis of the parallax information and coordinate positionof the pixel (unit area). The eye point of the grid map is set so as tobe higher than the eye point of the parallax image G1. The grid map isobtained by partitioning an area spreading ahead of the vehicle (anexample of an area around the vehicle) in a grid.

Subsequently, in step S105, the ECU 2 executes the process of outputtingtraveling road surface information (detection step) with the use of thetraveling road surface information output unit 14. The traveling roadsurface information output unit 14, for example, outputs the travelingroad surface information that an aggregate of road surface grids RGclosest to the vehicle is a traveling road surface. The traveling roadsurface information output unit 14, for example, outputs the travelingroad surface information to an ECU that automatically drives the vehicleor an ECU that supports driving of the vehicle. When step S105 ends, theECU 2 ends the flowchart shown in FIG. 7.

The details of the process of step S105 shown in FIG. 7 are shown inFIG. 8. FIG. 8 is a flowchart that shows an example of the process ofoutputting traveling road surface information by the traveling roadsurface detection apparatus 1.

As shown in FIG. 8, the ECU 2 executes the process of setting a searchID to be processed with the use of the traveling road surfaceinformation output unit 14 in step S200. The traveling road surfaceinformation output unit 14 selects and sets one search ID to beprocessed from among a plurality of search IDs.

Subsequently, in step S201, the ECU 2 executes the process of extractinggrids with the use of the traveling road surface information output unit14. The traveling road surface information output unit 14 extracts allthe intended grids on the grid map G2 on the basis of the search ID setin step S200.

Subsequently, in step S202, the ECU 2 executes the process of settinggrids to be processed with the use of the traveling road surfaceinformation output unit 14. The traveling road surface informationoutput unit 14 sets grids to be processed on the basis of the search IDset in step S200. For example, the traveling road surface informationoutput unit 14 arranges all the intended grids on the grid map G2 inorder of distance to the vehicle, and selects one grid to be processed.The traveling road surface information output unit 14 selects one gridsuch that a grid is preferentially processed as the distance to the gridreduces.

Subsequently, in step S204, the ECU 2 executes road surface griddetermination process with the use of the traveling road surfaceinformation output unit 14. The traveling road surface informationoutput unit 14 determines whether the grid set in step S202 is a roadsurface grid RG or a non-road surface grid NG. That is, the travelingroad surface information output unit 14 determines whether the grid setin step S202 is a gap grid EG. When the traveling road surfaceinformation output unit 14 determines that the grid set in step S202 isa road surface grid RG or a non-road surface grid NG (YES in S204), theECU 2 advances the process to step S207. That is, when the grid set instep S202 is not a gap grid, the ECU 2 advances the process to stepS207. On the other hand, when the traveling road surface informationoutput unit 14 determines that the grid set in step S202 is neither aroad surface grid RG nor a non-road surface grid NG (NO in S204), theECU 2 advances the process to step S205. That is, when the grid set instep S202 is a gap grid, the ECU 2 advances the process to step S205.

The ECU 2 executes same label determination process with the use of thetraveling road surface information output unit 14 in step S205. Thetraveling road surface information output unit 14 determines whether thegap grid EG that is the grid set in step S202 is located between roadsurface grids RG with the same label. When the traveling road surfaceinformation output unit 14 determines that the gap grid EG is locatedbetween road surface grids RG with the same label (YES in S205), the ECU2 advances the process to step S206. On the other hand, when thetraveling road surface information output unit 14 determines that thegap grid EG is not located between road surface grids RG with the samelabel (NO in S205), the ECU 2 advances the process to step S207.

The ECU 2 executes interpolation process with the use of the travelingroad surface information output unit 14 in step S206. The traveling roadsurface information output unit 14 changes the gap grid EG that is thegrid set in step S202 to a road surface grid RG. After that, the ECU 2advances the process to step S207.

The ECU 2 executes the process of determining whether the process hascompleted on all the grids extracted in step S201 with the use of thetraveling road surface information output unit 14 in step S207. When thetraveling road surface information output unit 14 determines that theprocess has not completed on all the grids (NO in S207), the ECU 2returns to step S202, and executes the process of setting the next gridto be processed. That is, until the traveling road surface informationoutput unit 14 determines that the process has completed on all thegrids extracted in step S201 (YES in S207), the ECU 2 repeatedlyexecutes step S202 to step S207. Thus, searching regarding one of thearrows shown in FIG. 6A completes, and the gap grid EG that satisfies acondition is changed to a road surface grid RG.

When the traveling road surface information output unit 14 determinesthat the process has completed on all the grids (YES in S207), the ECU 2executes the process of determining whether the process has completed onall the search IDs with the use of the traveling road surfaceinformation output unit 14. When the traveling road surface informationoutput unit 14 determines that the process has not completed on all thesearch IDs (NO in S208), the ECU 2 returns to step S200, and executesthe process of determining the next search ID to be processed. That is,until the traveling road surface information output unit 14 determinesthat the process has completed on all the search IDs (YES in S208), theECU 2 repeatedly executes step S200 to step S208. Thus, searchingregarding all the arrows shown in FIG. 6A completes.

Subsequently, in step S209, the ECU 2 executes the process of outputtingtraveling road surface information with the use of the traveling roadsurface information output unit 14. The traveling road surfaceinformation output unit 14 detects an aggregate obtained by connectingcontinuous road surface grids RG determined in each searching directionas a traveling road surface on the grid map G2. The traveling roadsurface information output unit 14 outputs information about thedetected traveling road surface (traveling road surface information) to,for example, another ECU, or the like. When the process of step S209ends, the flowchart shown in FIG. 8 ends.

Operation and Advantageous Effects of Traveling Road Surface DetectionApparatus and Traveling Road Surface Detection Method

With the above-described traveling road surface detection apparatus(traveling road surface detection method) 1, after the coordinatetransformation unit 13 performs coordinate transformation from theparallax image G1 to the grid map G2, each gap grid EG located betweenroad surface grids RG on the grid map G2 is detected as a traveling roadsurface on the grid map G2 by the traveling road surface informationoutput unit 14. Therefore, with the traveling road surface detectionapparatus 1, for example, even when areas on the grid map G2,corresponding to the road surface areas R1, R2, R3, are discontinuous atthe time when the parallax image G1 including the road surface areas R1,R2, R3 are subjected to coordinate transformation and projected onto thegrid map G2, it is possible to detect a traveling road surface byinterpolating discontinuous areas. Therefore, this apparatus is able toimprove a decrease in the accuracy of detecting a traveling roadsurface.

With the traveling road surface detection apparatus (traveling roadsurface detection method) 1, the traveling road surface informationoutput unit 14 is able to set each gap grid EG sandwiched between roadsurface grids RG arranged in one of radially extending directions withreference to the position of the stereo camera 3 on the grid map G2 forthe gap grid EG located between road surface grids RG on the grid mapG2. For example, if the apparatus is configured to detect each gap gridEG sandwiched between road surface grids RG arranged in the lateraldirection (vehicle width direction) with reference to the position ofthe stereo camera 3 on the grid map G2 as a traveling road surface,there is a concern that a gap grid EG behind an obstacle, which cannotbe recognized from the stereo camera 3 that is an observation point, isdetected as a traveling road surface. In contrast, the traveling roadsurface detection apparatus 1 scans in the radially extending directionswith reference to the position of the stereo camera 3 on the grid mapG2. Thus, the traveling road surface detection apparatus 1 sets each gapgrid EG sandwiched between road surface grids RG arranged in one of theradially extending directions with reference to the position of thestereo camera 3 for the gap grid EG located between road surface gridsRG on the grid map G2. Therefore, for example, the traveling roadsurface detection apparatus 1 is able to avoid erroneous detection of agap grid EG behind an obstacle as a traveling road surface.

With the traveling road surface detection apparatus (traveling roadsurface detection method) 1, the traveling road surface informationoutput unit 14 may be configured to, when labels respectively associatedwith pixels corresponding to a pair of the road surface grids RG thatsandwich the gap grid EG are different from each other, not detect thegap grid EG sandwiched between the pair of the road surface grids RG asa traveling road surface. In this case, for example, when the pluralityof road surface areas R1, R2, R3 are included in the parallax image G1,the traveling road surface detection apparatus 1 is able to avoiderroneous detection of a gap grid EG corresponding to a pixel betweenroad surface areas as a traveling road surface.

The embodiment of the invention is described above; however, theinvention is not limited to the above-described embodiment. For example,in the above-described embodiment, the example in which the ECU 2includes the image acquisition unit 10, the area determination unit 11,the area identification unit 12, the coordinate transformation unit 13and the traveling road surface information output unit (detection unit)14 is described. Instead, the ECU 2 does not necessarily include thearea identification unit 12.

In the above-described embodiment, the traveling road surface detectionapparatus 1 is described as an apparatus mounted on the vehicle;however, the traveling road surface detection apparatus 1 is not limitedto an in-vehicle apparatus. For example, the traveling road surfacedetection apparatus 1 may be a server, or the like, that is installedoutside the vehicle and that is able to acquire a parallax image throughcommunication.

In the above-described embodiment, the example in which the first camera4 and the second camera 5 are provided on the back side of thewindshield of the vehicle and capture images ahead of the vehicle;however, the installation position of the first camera 4 and secondcamera 5 is not limited to this position. For example, the first camera4 and the second camera 5 may be provided at the side of the vehicle orthe rear (for example, the back side of a rear window) of the vehicle,and may capture images to the side or rear of the vehicle. In this case,the coordinate transformation unit 13 just needs to prepare a grid mapsuch that the eye point of the grid map is set so as to be higher thanthe eye point of a parallax image and an area to the side or rear of thevehicle is partitioned in a grid, and then execute coordinatetransformation process. That is, the coordinate transformation unit 13just needs to prepare a grid map in which an area around the vehicle ispartitioned in a grid commensurately with a direction in which aparallax image is captured. With this configuration, for example, evenwhen the vehicle reverses, the traveling road surface detectionapparatus 1 is able to improve a decrease in the accuracy of detecting atraveling road surface.

In the above-described embodiment, the grid map G2 in plan view isdescribed; however, the grid map is not limited to a map in plan view.The grid map may be a bird's-eye view of which the eye point is set soas to be higher than the eye point of the parallax image G1. FIG. 5Bshows an example of the grid map G2 corresponding to the predeterminedarea A1 in the parallax image G1. The size of the predetermined area A1may be set as needed. For example, the overall parallax image G1 may beset for the predetermined area A1.

In the above-described embodiment, the example in which the travelingroad surface detection apparatus 1 is connected to the stereo camera 3is described.

Instead of the stereo camera 3, a monocular camera may be provided. Witha monocular camera, it is possible to obtain a parallax image by using aknown technique (for example, a technique using a time difference incapturing an image in the vehicle that is traveling).

In the above-described embodiment, the example in which the travelingroad surface information output unit 14 converts each gap grid EGlocated between road surface grids RG to a road surface grid RG isdescribed. However, it is not always necessary to execute the process ofconverting the gap grid EG to a road surface grid RG. A traveling roadsurface may be detected on the assumption that each gap grid EG is aroad surface grid RG at the time when connectivity of the road surfacegrids RG is finally determined. Such a process may be, for example,implemented by using a road surface grid assumption flag, or the like.

What is claimed is:
 1. A traveling road surface detection apparatus thatdetects a traveling road surface on the basis of a parallax image arounda vehicle, the parallax image being acquired by an in-vehicle camera,the traveling road surface detection apparatus comprising: an electroniccontrol unit configured to i) determine on the basis of parallaxinformation of the parallax image whether a unit area is a road surfacearea or a non-road surface area for each unit area of the parallaximage, each unit area including one or multiple pixels, ii) associateeach unit area with a grid in a map on the basis of parallaxinformation, and coordinate position of the unit area, an eye point ofthe map being set so as to be higher than an eye point of the parallaximage, the map being obtained by partitioning an area around the vehiclein a grid, and iii) detect road surface grids and each grid locatedbetween the road surface grids on the map as the traveling road surfaceon the map, the road surface grids being the grids corresponding to theunit areas determined as the road surface areas.
 2. The traveling roadsurface detection apparatus according to claim 1, wherein the electroniccontrol unit detects the traveling road surface on the map by settingeach grid sandwiched between the road surface grids arranged in one ofradially extending directions with reference to a position of thein-vehicle camera on the map for the grid located between the roadsurface grids on the map.
 3. The traveling road surface detectionapparatus according to claim 1, wherein: the electronic control unitconfigured to associate identification information that identifies thecorresponding road surface area with each unit area determined as theroad surface area, on the basis of a coordinate position or pixelinformation of each unit area determined as the road surface areawherein when the pieces of identification information respectivelyassociated with the unit areas corresponding to a pair of the roadsurface grids that sandwich the grid are different from each other, theelectronic control unit does not detect the grid sandwiched between thepair of the road surface grids as the traveling road surface.
 4. Atraveling road surface detection method that detects a traveling roadsurface on the basis of a parallax image around a vehicle including anelectronic control unit, the parallax image being acquired by anin-vehicle camera, the traveling road surface detection methodcomprising: determining, by the electronic control unit, on the basis ofparallax information of the parallax image whether a unit area is a roadsurface area or a non-road surface area for each unit area of theparallax image, each unit area including one or multiple pixels;associating, by the electronic control unit, each unit area with a gridin a map on the basis of parallax information and coordinate position ofthe unit area, an eye point of the map being set so as to be higher thanan eye point of the parallax image, the map being obtained bypartitioning an area around the vehicle in a grid; and detecting, by theelectronic control unit, road surface grids and each grid locatedbetween the road surface grids on the map as the traveling road surfaceon the map, the road surface grids being the grids corresponding to theunit areas determined as the road surface areas.